Content item audience selection

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for selecting an audience for content are disclosed. In one aspect, a method includes receiving, from a content item provider, a request to distribute content items to users that have been deemed interested in a particular entity. First users that have expressed an interest in the particular entity are identified. Expansion entities for the particular entity are identified in a knowledge graph. At least one of the expansion entities can be connected to the particular entity by a relationship path. Second users are identified. The second users are deemed interested in the particular entity based on the second users having expressed an interest in an expansion entity. The content items are provided to at least a portion of the first users and at least a portion of the second users.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. application Ser. No.15/860,830, filed Jan. 3, 2018, which is a continuation of U.S.application Ser. No. 13/790,054, filed Mar. 8, 2013, the contents ofwhich are incorporated by reference herein.

BACKGROUND

This specification relates to data processing and audience selection.

The Internet provides access to a wide variety of resources. Forexample, video and/or audio files, as well as web pages for particularsubjects or that present particular news articles are accessible overthe Internet. Access to these resources presents opportunities foradvertisements (or other sponsored content items) to be provided withthe resources. For example, a web page can include “slots” (i.e.,specified portions of the web page) in which advertisements (or othercontent items) can be presented. These slots can be defined in the webpage or defined for presentation with a web page, for example, in aseparate browser window. Advertisements or other content items that arepresented in slots of a resource are selected for presentation by acontent distribution system.

SUMMARY

This document describes methods, systems, and computer readable mediumthat facilitate selection of an audience to which one or more contentitems, such as advertisements, will be provided. The audience can beselected based on distribution criteria that are provided by a contentitem provider for which the content item is being distributed. Thedistribution criteria can specify, for example, a desired reachindicating that the content item should be distributed to a specifiednumber (e.g., 1,000,000) of interested users for a particular entity(e.g., users that have been deemed interested in a particularidentifiable person (e.g., a celebrity), place (e.g., national park),thing (e.g., ice cream), or concept (e.g., biology)).

For example, an advertiser that is promoting a science fiction movie mayspecify that the advertisement for the new science fiction movie bepresented to interested users for a prequel to the science fiction movie(e.g., users that have been deemed interested in the prequel). In turn,a content item distribution system can identify, as directly interestedusers for the prequel, users that have expressed an interest in theprequel (e.g., through social networking posts or interaction with apositive feedback user interface element). These directly interestedusers can then be considered audience members for the advertisement forthe new movie.

In some situations, the number of directly interested users for aparticular entity (e.g., the prequel entity in the example above), willbe less than the desired reach for the content item. In thesesituations, additional audience members can be selected by identifyingusers that are likely to be interested in the particular entity based ontheir expressed interest in another entity that has been identified ashaving a particular relationship with the particular entity. Forexample, users that have not explicitly expressed an interest in theprequel may still become additional audience members based on theirexpressed interest in other movies that were directed by the director ofthe prequel, their expressed interest in actors that starred in theprequel, or their expressed interest in other entities that have anidentified relationship with the prequel.

In some implementations, selection of additional audience members for acontent item is performed using information from a social graph andinformation from a knowledge graph (e.g., a graph different than thesocial graph). For example, using the social graph, other users that areconnected to the directly interested users can be identified, and otherentities in which the other users have expressed an interest can bedetermined. One or more of these other entities can be selected asexpansion entities for the particular entity based, at least in part, ona number of the other users that have expressed an interest in the oneor more other entities and/or relationships between the one or moreentities and the particular entity. For example, assume that a thresholdportion of the other users expressed an interest in one of the actors ofthe prequel. In this example, the expansion entities for the prequel maybe the actors of the prequel (or other movies in which the actorsstarred), and interested users for any of the actors (or the othermovies) may be identified as additional audience members for the contentitem, thereby expanding the audience for the content item based on therelationships between users in a social graph and relationships betweenentities in a knowledge graph.

In situations in which the systems discussed herein collect informationabout users, or may make use of information about users, the users maybe provided with an opportunity to control whether programs or featurescollect user information (e.g., information about a user's socialnetwork, social actions or activities, profession, a user's preferences,or a user's current location), or to control whether and/or how toreceive content from the content server that may be more relevant to theuser. In addition, certain data may be treated in one or more waysbefore it is stored or used. For example, a user's identity may betreated so that no identifying information can be determined for theuser, or a user's geographical location may be generalized wherelocation information is obtained (such as to a city, ZIP code, or statelevel), so that a particular location of a user cannot be determined.Thus, the user may have control over how information is collected aboutthe user and used by a content server.

In general, one innovative aspect of the subject matter described inthis specification can be embodied in methods that include the actionsof receiving a request to distribute content to an audience of usersthat have been deemed interested in a particular entity; identifyingdirectly interested users that have expressed an interest in theparticular entity; identifying socially connected users that areconnected, in a social graph, to the directly interested users;identifying, in a knowledge graph, candidate entities that arereferenced by interests of the connected users; identifying, for each ofone or more of the candidate entities and based on the knowledge graph,a matching relationship between the candidate entity and the particularentity, the matching relationship being a relationship that each of thecandidate entity and particular entity share with a same entity;determining a relationship score for each matching relationship, therelationship score for the matching relationship being determined basedon a portion of the connected users that have expressed an interest inan entity having the matching relationship with the same entity;selecting an expansion relationship for the particular entity, theexpansion relationship being selected based on relationship scores formatching relationships; identifying, based on the social graph, a set ofadditional users that have expressed an interest in an entity having theexpansion relationship; and selecting, as an audience for the content,at least a portion of the directly interested users and at least aportion of the set of additional users. Other embodiments of this aspectinclude corresponding systems, apparatus, and computer programs,configured to perform the actions of the methods, encoded on computerstorage devices.

These and other embodiments can each optionally include one or more ofthe following features. Determining a relationship score for eachmatching relationship comprises, for each of at least one matchingrelationship: identifying a portion of the socially connected users thatare directly interested users for an entity having the matchingrelationship; identifying a portion of the candidate entities having thematching relationship; and determining the relationship score based, atleast in part, on the identified portion of the socially connected usersand the identified portion of the candidate entities.

Methods can further include the actions of obtaining a performancemeasure for the expansion relationship, the performance measure beingbased on a number of user interactions with the content that wasprovided to users that have expressed an interest in entities having theexpansion relationship; and updating the relationship score for theexpansion relationship based on the obtained performance measures.

Methods can further include the actions of receiving bid data specifyinga first value that will be paid by a content item provider fordistribution of the content to users that have been deemed interested inthe particular entity; and determining, based on the bid data, a secondvalue that the content item provider will pay for distribution of thecontent to the set of additional users. Determining the second value caninclude determining an expansion bid for the expansion relationship, theexpansion bid value being determined based, at least in part, on therelationship score for the expansion relationship, the expansion bidvalue being different from a bid value for distribution of the contentitem to the directly interested users. Identifying directly interestedusers can include identifying users that have interacted with a positivefeedback user interface element associated with content referencing theparticular entity.

Particular embodiments of the subject matter described in thisspecification can be implemented so as to realize one or more of thefollowing advantages. Implicit interests of users can be determinedbased on the explicit interests specified by the users and/or socialconnections of the users. The implicit interests can be used to identifyadditional audience members for a content item when the number ofexplicitly interested users does not meet a desired reach for thecontent item.

The details of one or more embodiments of the subject matter describedin this specification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an example environment in which content isdistributed to user devices.

FIG. 2 is a block diagram of an example data flow for selecting anaudience.

FIG. 3 is a block diagram of an example process for selecting audiencemembers for a content item.

FIG. 4 is a block diagram of another example process for selectingaudience members for a content item.

FIG. 5 is a flow chart of an example process for distributing contentbased on an entity bid.

FIG. 6 is block diagram of an example computer system.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

FIG. 1 is a block diagram of an example environment 100 in which contentis distributed to user devices 106. The example environment 100 includesa network 102, such as a local area network (LAN), a wide area network(WAN), the Internet, or a combination thereof. The network 102 connectswebsites 104, user devices 106, advertisers 108, and a contentdistribution system 110. The example environment 100 may include manydifferent websites 104, user devices 106, and advertisers 108.

A website 104 is one or more resources 105 associated with a domain nameand hosted by one or more servers. An example website is a collection ofweb pages formatted in hypertext markup language (HTML) that can containtext, images, multimedia content, and programming elements, such asscripts. Each website 104 is maintained by a publisher, which is anentity that controls, manages and/or owns the website 104.

A resource 105 is any data that can be provided over the network 102. Aresource 105 is identified by a resource address that is associated withthe resource 105. Resources include HTML pages, word processingdocuments, and portable document format (PDF) documents, images, video,and feed sources, to name only a few. The resources can include content,such as words, phrases, images and sounds, that may include embeddedinformation (such as meta-information in hyperlinks) and/or embeddedinstructions (such as scripts). Units of content that are presented in(or with) resources are referred to as content items.

A user device 106 is an electronic device that is capable of requestingand receiving resources over the network 102. Example user devices 106include personal computers, mobile communication devices, and otherdevices that can send and receive data over the network 102. A userdevice 106 typically includes a user application, such as a web browser,to facilitate the sending and receiving of data over the network 102.

A user device 106 can submit a resource request 112 that requests aresource 105 from a website 104. In turn, data representing therequested resource 114 can be provided to the user device 106 forpresentation by the user device 106. The requested resource 114 can be,for example, a home page of a website 104, web page from a socialnetwork, or another resource 105. The data representing the requestedresource 114 can include data that cause presentation of resourcecontent 116 at the user device 106. The data representing the requestedresource 114 can also include data specifying a portion of the resource(e.g., a portion of a web page) or a portion of a user display (e.g., apresentation location of another window or in a slot of a web page) inwhich content items, such as advertisements, can be presented.Throughout this document, these specified portions of the resource oruser display are referred to interchangeably as content items slots oradvertisement slots 118. Any type of content (e.g., content items otherthan advertisements) can be presented in these advertisement slots 118.

When a resource 105 is requested by a user device 106, execution of codeassociated with an advertisement slot 118 in the resource initiates arequest for an advertisement (or another type of content item) topopulate the advertisement slot 118. When a resource requests anadvertisement (or another content item), the resource is considered tohave provided an impression. As used throughout this document, the termimpression refers to a presentation opportunity for a content item.Impressions are considered to be allocated to advertisers (or othercontent item providers) that are selected to provide an advertisement(or another content item for presentation.

In some situations, impressions are allocated by a content distributionsystem 110. For example, some publishers enter into an agreement to haveadvertisement slots 118 on their resources 105 populated byadvertisements (or other content items) selected by the contentdistribution system 110. In these situations, the publisher willgenerally include, in the advertisement slots 118, code that, whenexecuted by the user device 106, submits an advertisement request to thecontent distribution system 110.

The advertisement request can include characteristics of theadvertisement slots 118 that are defined for the requested resource 114.For example, a reference (e.g., URL) to the requested resource 114 forwhich the advertisement slot 118 is defined, a size of the advertisementslot 118, and/or media types that are eligible for presentation in theadvertisement slot 118 can be provided to the content distributionsystem 110. Similarly, keywords associated with a requested resource(“resource keywords”) or entities that are referenced by the resourcecan also be provided to the content distribution system 110 tofacilitate identification of advertisements that are relevant to therequested resource 114.

The advertisements (or other content items) that are provided inresponse to an advertisement request (or another content item request)are selected based on distribution criteria for the advertisements.Distribution criteria are a set of criteria upon which distribution ofcontent items are conditioned. In some implementations, the distributioncriteria for a particular advertisement (or other content item) caninclude distribution keywords that must be matched (e.g., by resourcekeywords) in order for the advertisement to be eligible forpresentation. The distribution criteria can also specify a bid and/orbudget for distributing the particular advertisement. Bids are used toconduct an auction to select which advertisement(s) will be presentedand/or in which advertisement slot the advertisement(s) will bepresented. A content item provider can specify a budget, which willlimit the maximum amount that the content item provider will spend overa specified period.

In some implementations, the distribution criteria can also specify aminimum number of users that should be provided the particularadvertisement, which is referred to as the reach of the advertisement.For example, the advertiser can specify that a particular advertisementshould be presented to at least 1,000,000 different users, such that thedesired reach for the advertisement is 1,000,000 unique users. In thisexample, the content distribution system can identify users that qualifyto receive the advertisement (e.g., based on the distribution criteria),and when a page is presented to these qualified users the particularadvertisement can be presented in an advertising slot in the page.

In some implementations, the distribution criteria for a particularadvertisement (or group of advertisements) can further specify that theadvertisement be presented to users that have been deemed interested ina particular entity. A user can be deemed interested in a particularentity, for example, based on the user affirmatively expressing aninterest in the particular entity. For example, a user can be deemedinterested in a particular movie based on the user publishing a socialnetwork post referencing the particular movie, interacting with apositive feedback element associated with the particular movie, orvisiting a resource about the particular movie. A user that hasaffirmatively expressed an interest in a particular entity is referredto as a directly interested user for the particular entity.

When the number of directly interested users that have explicitlyexpressed an interest in the particular entity meets the desired reachfor an advertisement, the desired reach specified by the advertiser canbe met by providing the advertisement to this selected audience. Thus,this set of directly interested users is the only audience members thatneed to be selected for the advertisement to meet the desired reach forthe advertisement.

When the number of directly interested users for the particular entitydoes not meet the desired reach for the advertisement, the desired reachfor the advertisement may still be met by selecting at least some of thedirectly interested users as a portion of the audience members for theadvertisement and identifying additional audience members for theadvertisement. As described in more detail below, the additionalaudience members can include users that are considered interested in theparticular entity based on their expressed interest in other entities.

The environment 100 includes an audience selection apparatus 120 thatcan identify additional audience members for an advertisement (oranother content item). The audience selection apparatus 120 is a dataprocessing apparatus that utilizes social connections between users andrelationship paths between entities to identify users that are likely tobe interested in a particular entity, and therefore considered to beinterested in the particular entity, even if the users have notexplicitly expressed an interest in the particular entity. To facilitateidentification of additional audience members for the particular entity,the audience selection apparatus 120 can use information from a socialgraph 122 and a knowledge graph 124, which are each described in moredetail with reference to FIG. 2.

In some implementations, the audience selection apparatus 120 can accessor otherwise reference a social graph 122 to identify other users thatare considered similar to the directly interested users for theparticular entity, and identify other entities in which the other usershave expressed an interest. For example, the audience selectionapparatus 120 can identify, from the social graph 122, the users thatare socially connected (e.g., by way of acceptance of an invitation tobe socially connected) to the interested users that have expressed aninterest in the particular entity, and identify or log the otherentities in which these identified users expressed an interest. Theidentified or logged entities are referred to as candidate entities.Users that are socially connected to a directly interested user arereferred to as socially connected users (or connected users).

The audience selection apparatus 120 accesses or otherwise references aknowledge graph 124 to identify relationships between the candidateentities and the particular entity. For example, some portion ofcandidate entities may be identified in the knowledge graph 124 asactors in the prequel from the example above, while other candidateentities may be other movies that have a same director as the prequel.

As described in more detail below, the audience selection apparatus 120can use the identified relationships between the candidate entities andthe particular entity to identify an expansion entity (or set ofexpansion entities) or an expansion relationship for the particularentity. An expansion entity is an entity, other than the particularentity, that shares a relationship in the knowledge graph with theparticular entity. The expansion entity can be used to expand anaudience to whom a content item will be provided when a content itemprovider has requested the content item be provided to users interestedin the particular entity.

For example, assume that at least a portion of the socially connectedusers expressed an interest in movies that had a same director as theprequel from the example above. In this example, the audience selectionapparatus 120 may determine that users who have expressed an interest inmovies having the same director as the prequel are likely to also beinterested in the prequel even if they have not explicitly expressed aninterest in the prequel. Therefore, the audience selection apparatus 120can select movies having a same director as the prequel as expansionentities for the prequel. In turn, the audience selection apparatus 120can identify the users who have expressed an interest in movies by thesame screenwriter as additional users that are interested in the prequeland select these identified users as additional audience members for theadvertisement (e.g., users beyond the directly interested users for theprequel).

FIG. 2 is a block diagram of an example data flow 200 for selecting anaudience. The data flow 200 begins with the audience selection apparatus120 receiving distribution criteria 202 for an advertiser from thecontent distribution system 110 (or a data store). As discussed above,the distribution criteria 202 can include a desired audience reach foran advertisement and a particular entity (or set of entities) that hasbeen determined to be of interest to the audience members. For example,the distribution criteria 202 for the advertiser may specify that theadvertiser is requesting a desired audience reach of 1,000,000 users,and that the desired audience is a set of users that are interested in aprequel of a movie that is being promoted by the advertisement. Asdescribed with reference to FIG. 5, the distribution criteria can alsoinclude bid data for the content item.

The audience selection apparatus 120 can access a social graph 204(e.g., stored in a data store) to identify directly interested users forthe particular entity (e.g., users that have expressed an interest inthe particular entity specified by the distribution criteria). A socialgraph is a representation of social links between users and can includeinformation that the users have provided in a social networkingenvironment (e.g., topics of interest). For example, a social graph canrepresent interpersonal relationships between two or more differentusers. Each node in a social graph represents a particular user andlinks that connect two nodes indicate that users represented by the twonodes are socially related (e.g., through a social networkrelationship).

Two users can be socially related, for example, by mutually assenting tobe socially related to each other (e.g., in a social networkingenvironment). For example, one user (e.g., UserA 206) can send anotheruser (e.g., UserD 208) a request to be socially related and the userthat receives the request (e.g., UserD 208) can affirm (e.g., byaccepting the request) that a social relation between the users exists.In turn, a link (e.g., link 210) can be created between the nodes thatrepresent the two users.

FIG. 2 provides a visual representation of the social graph 204, but thesocial graph 204 can also be represented in other ways. For example, thesocial graph 204 can be stored as an indexed set of relationshipsbetween nodes that can be accessed to identify relationships between aparticular node and other nodes. The nodes of the social graph 204 canalso be indexed according to their respective interests to facilitateidentification of users that have expressed an interest in a particularentity.

As described above, users can express an interest in a particularentity, for example, by providing data explicitly indicating that theuser is interested in the particular entity. For example, a user thatliked the prequel from the example above may specify in their socialprofile or through a post in a shared data stream that they liked theprequel. The user may also explicitly express an interest in the prequelby visiting a social profile page for the prequel and interact with apositive feedback user interface element (e.g., a button used to expressa positive interest in an entity). Further the user may explicitlyexpress an interest in the prequel by interacting with a positivefeedback user interface element included in a post made by another user.

Continuing with the example above, the audience selection apparatus 120can identify a set of directly interested users 212 that have explicitlyexpressed an interest in the particular entity using the social graph.For example, based on the social graph 204, the audience selectionapparatus 120 can determine that that UserA 206 and UserB 210 have eachexplicitly expressed an interest in the prequel, such that UserA 206 andUserB 210 are each considered directly interested users for the prequel.In turn, the set of directly interested users 212 can be identified as aset of audience members for the advertisement, assuming that any otherdistribution criteria are also met for each user in the set. In thisexample, if the audience selection apparatus 120 determines that thenumber of users in the set of directly interested users 212 meets thedesired reach for the advertisement for the movie, the set of directlyinterested users 212 can be selected as the complete audience for theadvertisement.

The audience selection apparatus 120 can also include additional users(e.g., users that have not explicitly expressed an interest in theparticular entity) in the audience for the advertisement. For example,if the number of directly interested users that have explicitlyexpressed an interest in the particular entity is less than the desiredreach for the advertisement, the audience selection apparatus 120 mayidentify additional users to be included in the audience with thedirectly interested users 212. Similarly, even when the number ofdirectly interested users for the particular entity meets the desiredreach for the advertisement, the audience selection apparatus 120 caninclude additional users in the audience for an advertisement to extendthe reach of the advertisement. The additional users that are includedin the audience for a content item are referred to as additionalaudience members.

Additional audience members for a particular entity may be users thathave not yet been deemed to have explicitly expressed an interest in theparticular entity, but are still considered interested in the entitybased on information derived, in part, from the social graph 204 and/ora knowledge graph 214. For brevity, additional users that are consideredinterested in a particular entity without having been deemed to haveexplicitly expressed an interest in the particular entity are referredto as implicitly interested users for the particular entity.

To identify implicitly interested users for a particular entity, theaudience selection apparatus 120 can identify other entities of interestto other users that are socially connected to directly interested users,determine, among the other users, a level of interest in the otherentities, and/or evaluate the relationships between the other entitiesand the particular entity. In turn, the audience selection apparatus 120can select an expansion entity from the other entities, and identify, asimplicitly interested users for the particular entity, users that aredirectly interested users for the expansion entity.

For example, with reference to FIG. 2, the audience selection apparatus120 can access (or otherwise obtain information from) the social graph204 to determine that UserA 206 is socially connected to UserD 208 andUserE 216, and that UserB 210 is socially connected to UserE 216 andUserC 218. Thus, the audience selection apparatus 120 can identify UserC218, UserD 208, and UserE 216 are members of a set of connected users220 that are socially connected to the directly interested users for theparticular entity.

The audience selection apparatus 120 can identify a set of candidateentities 222 based on the interests for the set of connected users 220.The set of candidate entities 222 for the set of connected users 220 arethose entities for which at least one user from the set of users 220 isa directly interested user. For example, UserD 208 has explicitlyspecified an interest in MovieA. Therefore, the audience selectionapparatus 120 can include MovieA in the set of candidate entities 222.Similarly, UserE 216 has explicitly expressed an interest in MovieB,while UserC has explicitly expressed an interest in MovieC, such thatthe audience selection apparatus 120 can include both MovieB and MovieCin the set of candidate entities 222. In this example, each of UserD andUserE are both directly interested users for DirectorA. Thus, DirectorAis also included in the set of candidate entities 222.

In some implementations, the audience selection apparatus 120 accesses(or otherwise obtains information from) a knowledge graph 214 toidentify relationships between the candidate entities in the set ofcandidate entities 222 and the particular entity, and uses theserelationships to facilitate identification of one or more expansionentities for the particular entity. As used in this document, aknowledge graph is a representation of relationships between uniqueentities, and the knowledge graph can be stored in one or more datastores. Each node in the knowledge graph represents a different entityand pairs of nodes in the knowledge graph are connected by relationshippaths (e.g., graph edges) that indicate a relationship between the twoentities represented by the pair of nodes.

For example, the knowledge graph 214 includes node 224 representing theprequel 224, node 226 representing ActorA, node 228 representingDirectorA, nodes 230, 232, and 234 that respectively represent MovieA,MovieB, and MovieC, and node 236 representing ActressA. Node 224 andnode 226 are connected by a relationship path 238 indicating that ActorAis an actor in the prequel. Node 228, representing DirectorA, isconnected to each of node 224, 230, 232, and 234 by relationship paths240, 242, 244, and 246 indicating that DirectorA is the director of eachof the prequel, MovieA, MovieB, and MovieC.

A single pair of nodes can have multiple different relationship pathsthat connect the pair of nodes. For example, node 236, which representsActressA, is connected to node 228, which represents DirectorA, byrelationship path 248 indicating that ActressA is the daughter ofDirectorA, and by relationship path 250 indicating that DirectorA is thefather of ActressA. If additional relationships were identified betweenDirectorA and ActressA additional relationship paths could be used torepresent these relationships.

Using information obtained from the knowledge graph 214 and informationobtained from the social graph 204, the audience selection apparatus 120can generate a set of interest scores 252 (e.g., Int_ScoreX; Int_ScoreY;Int_ScoreZ). An interest score is a measure of interest in an entity. Insome implementations, the interest score indicates a likelihood thatdirectly interested users for a candidate entity are also interestedusers for the particular entity. When the interest score for a candidateentity is above a specified threshold, the candidate entity can beidentified as an expansion entity for the particular entity. In turn,directly interested users for the expansion entity are classified asimplicitly interested users for the particular entity.

Interest scores can be determined, based at least in part, on a portion(e.g., a number or a percentage) of the users from the set of connectedusers 220 that are directly interested users for the candidate entity.For example, according to the social graph 204, UserD 208 and UserE 216are each directly interested users for DirectorA, while User C 218 isnot an explicitly interested user for DirectorA. In this example, 2 outof the 3 (e.g., 66.67% of) users in the set of connected users aredirectly interested users for DirectorA. Meanwhile, from the set ofconnected users 220, only UserD 208 is a directly interested user forMovieA, only UserE 216 is a directly interested user for MovieB, andonly UserC 218 is a directly interested user for MovieC. Among the setof connected users 220, DirectorA is the entity that has the highestportion of directly interested users. Thus, DirectorA could be anexpansion entity for the prequel if this portion of directly interestedusers were used as the interest score.

The interest scores can be based on additional (or different) factorsbeyond a portion of the set of connected users 220 that are directlyinterested users for the candidate entity. For example, the interestscore for a candidate entity can be increased when one or more of thedirectly interested users for the particular entity are also directlyinterested users for the candidate entity. For example, UserA 206, whois a directly interested user for the prequel is also a directlyinterested user for DirectorA, such that the interest score forDirectorA may be increased to reflect the shared interest of theserelated entities.

In some implementations, the interest score for a candidate entity isbased, at least in part, on a relationship score for matchingrelationships between the particular entity and the candidate entities.A relationship score is a measure of interest, e.g., by the set ofconnected users, in candidate entities having a particular relationshipthat is shared by the particular entity. For example, as noted above,the prequel has a relationship of “directed by” with DirectorA asindicated by the relationship path between the prequel and DirectorA.Similarly, MovieA, MovieB, and MovieC each have the relationship“directed by” with DirectorA. In this example, the relationship“directed by DirectorA” is a matching relationship between theparticular entity and the candidate entities MovieA, MovieB, and MovieC.The relationship score for the relationship “directed by DirectorA” canbe determined, for example, based on a portion of the set of connectedusers 220 that are directly interested users for other entities (e.g.,movies) that have the matching relationship “directed by DirectorA”(e.g., by being connected to DirectorA by way of a relationship pathspecifying the relationship “directed by”). For example, therelationship score for “directed by DirectorA” can increase as theportion of the set of users that are directly interested users for otherentities (e.g., movies) that are directed by Director A.

The relationship score can be used, for example, to increase (orotherwise adjust) the interest scores of entities that have the matchingrelationship. For example, the interest scores for MovieA, MovieB, andMovieC may be increased (e.g., scaled up) using the relationship scorefor “directed by Director A” since each of these entities has thematching relationship “directed by” with DirectorA. A relationship scorecan be determined for any relationship or set of relationships relativeto the particular entity.

The interest score for a candidate entity can be adjusted based on thedegree of separation, in the knowledge graph, between the candidateentity and the particular entity. For example, DirectorA has a firstdegree of separation with the prequel (e.g., because the node 228 isdirectly connected to the node 224 by relationship path 240), whileMovieB has a third degree of separation with the prequel (e.g., becausethe node 232 is separated from node 224 by two relationship paths).Thus, the interest score for DirectorA may be increased relative to theinterest score for MovieB (or the interest score for MovieB may bedecreased relative to the interest score for DirectorA) based onDirectorA having a lower degree of separation from the prequel.

In some implementations, the audience selection apparatus 120 selects anexpansion entity 254 for the particular entity based on the set ofinterest scores 252. For example, the audience selection apparatus 120can identify, as the expansion entity 254, the candidate entity havingthe highest interest score.

In some implementations, the audience selection apparatus 120 canadditionally, or alternatively, select an expansion relationship thatwill be used to select expansion entities. For example, the audienceselection apparatus 120 can identify the matching relationship having ahighest relationship score, and select, as expansion entities, one ormore entities (e.g., candidate entities or other entities) having thematching relationship. To illustrate, assume that the matchingrelationship “directed by DirectorA” is identified to have the highestrelationship score among all matching relationships. In this example,this matching relationship is selected as the expansion relationship.Thus, the audience selection apparatus 120 can select, as expansionentities for the particular entity, any (or all) entities having thematching relationship “directed by DirectorA,” even if some of theentities were not originally selected as candidate entities based on theinterests of the set of connected users 220. In some implementations, anexpansion relationship is a relationship representing a set of entitiesthat are likely to be of interest to a user that has expressed aninterest in any of the entities in the set. Expansion relationships aredescribed in more detail with reference to FIG. 4.

Using the social graph 204, the audience selection apparatus 120 canidentify users that are directly interested users for the expansionentities, and select these users (or a proper subset thereof) asadditional audience members for the advertisement. In turn, the contentdistribution system 110 can distribute the advertisement to at least aportion of the additional audience members.

FIG. 3 is a block diagram of an example process 300 for selectingaudience members for a content item. The process 300 can be performed byone or more data processing apparatus, such as the audience selectionapparatus 120 and/or the content distribution system 110 of FIG. 1.Operations of the process 300 can be implemented by execution ofinstructions stored on a non-transitory computer readable medium andthat cause one or more data processing apparatus to perform operationsof the process 300.

A request to distribute content to an audience of users that have beendeemed interested in a particular entity is received (302). In someimplementations, the request is a request to distribute one or moreadvertisements to an audience of users that are considered interested inthe particular entity. For example, as discussed above, an advertiser ofa new movie can submit a request to an advertisement management systemrequesting that the advertisement management system distribute anadvertisement for the new movie to users that are interested in aprequel of the movie.

The request can include, for example, a desired reach indicating anumber of users that are to be included in the audience of users to whomthe content is distributed. The desired reach can be expressed as atotal number of different users to whom the content is to be provided.For example, an advertiser that is promoting a new movie may requestthat an advertisement for the movie be presented to at least 1,000,000different users. Similarly, an advertiser that is promoting a new songmay request that a sample portion of the song be distributed to1,000,000 different users.

The particular entity can be any person, place, or concept specified bythe advertiser. For example, the advertiser that is promoting the newmovie may specify that the advertisement for the movie is to bepresented to users that have been deemed interested in a prequel of themovie. Similarly, the advertiser promoting the new song may request thatthe sample portion of the song be distributed to users that have beendeemed interested in the band that performs the song.

In some implementations, a set of expansion entities can be optionallyidentified in the knowledge graph based on the particular entity (303).The set of expansion entities can be identified as one or more entitiesthat are connected, either directly or indirectly, to the particularentity though a relationship path. For example, entities that aredirectly connected to the particular entity by one or more relationshippaths can be selected for inclusion in the set of expansion entities.Additionally, if the particular entity and another entity share a samerelationship with a third entity (e.g., where the third entity isdirectly connected to each of the particular entity and the otherentity), can be selected for inclusion in the set of expansion entities.The set of expansion entities can be used to select additional audiencemembers (314), as described below.

Directly interested users for the particular entity are identified(304). As noted above, directly interested users for a particular entityare those users that have expressed an interest in the particularentity. For example, a directly interested user for the prequel of themovie being advertised can be a user that has affirmatively expressed aninterest in the prequel. Similarly, a directly interested user for theband that performs the song can be a user that has affirmativelyexpressed an interest in the band.

A user can affirmatively express an interest in an entity, such as theprequel or the band discuss above, in many different ways. For example,a user can affirmatively express an interest in an entity by including areference to (e.g., including the name of) the entity in a socialnetwork post, interacting with a positive feedback element (e.g., a +1button) associated with a reference to the entity, or visiting a socialnetwork page about the entity. A user can also affirmatively express aninterest in an entity by identifying the entity as an interest (e.g., inan interests section of the user's social network profile).

Users that are socially connected to the directly interested users forthe particular entity are identified (306). In some implementations, thesocially connected users are users that are connected, in a socialgraph, to the directly interested users. For example, as described abovewith reference to FIG. 2, two users that have affirmatively identifiedeach other as a social connection (e.g., friend, family member, orbusiness colleague) can be represented by two nodes that are connectedby a link representing the social connection.

Using the social graph, a particular node representing a directlyinterested user can be identified, and each user that is represented bya node connected to the particular node can be deemed a sociallyconnected user (also referred to as a connected user) for the directlyinterested user. In this way, connected users for each of the directlyinterested users can be identified, and included in a set of sociallyconnected users for the directly interested users.

In some implementations, the identification of the socially connectedusers is performed in response to determining that the number ofdirectly interested users for the particular entity is less than thedesired reach that a content provider (e.g., an advertiser) hasspecified in the request to distribute content. For example, assume thatan advertiser has specified a reach of 1,000,000 users for a particularadvertisement, and that the users to whom the advertisement is presentedshould be users that have been deemed interested in a particular entity.In this example, if only 500,000 directly interested users for theparticular entity are identified, the socially connected users may thenbe identified as part of an audience selection process. Thus, in someimplementations, a determination that the number of directly interestedusers is less than the desired reach (e.g., a specified number of users)is made prior to identifying the socially connected users.

Candidate entities are identified based on the interests of the sociallyconnected users (308). In some implementations, the interests of eachsocially connected user (or a proper subset of the socially connectedusers) are identified from an interests section of the sociallyconnected user's social profile. In turn, a knowledge graph can be usedto identify, based on the interests of the socially connected users, aset of candidate expansion entities for the particular entity.

For example, assume that one socially connected user specifies, in aninterests section of their social profile, that they liked a particularmovie that is directed by the director that also directed the moviebeing advertised in the example above and the prequel. In this example,a knowledge graph can be used to identify a node representing thedirector, and the director can be added to a set of candidate expansionentities for the particular entity (also referred to as a candidateentity).

Similarly, assume that another socially connected user specified, in aninterests section of their social profile, that they liked a particularmusician that produced the soundtrack for the prequel. In this example,the knowledge graph can be used to identify the node representing themusician, and the musician can be added to the set of candidateentities.

In some implementations, the interests of the socially connected users,and therefore the candidate entities, can be identified based on otherinformation indicative of the interests of the socially connected users.For example, as noted above, interests of a socially connected user canbe identified based on the socially connected user's interactions withpositive feedback elements (e.g., endorsement buttons) that areassociated with entities, posts made by the socially connected user, andother online activity (e.g., registering to receive automatic updatesfrom a site that provides information about an entity). Any, or all, ofthe entities in which the socially connected users are deemed to haveexpressed an interest can be included in the set of candidate entitiesfor the particular entity.

An interest score is determined for each of the candidate entities(310). As noted above, an interest score is a measure of interest in anentity. In some implementations, the interest score indicates alikelihood that directly interested users in a candidate entity are alsointerested in the particular entity. Since socially connected users havean acknowledged social relationship, two socially connected users areconsidered more likely to have similar entity interests than tworandomly selected users. The similarity of interests between thesocially connected users and the directly interested users can befurther evidenced through similarities between the expressed interestsof the socially connected users and/or the directly interested users.

For example, even without additional information, the socially connectedusers can be assumed to have at least an initial level of interest inthe particular entity based on their social relationship with a directlyinterested user for the particular entity. Additionally, the level ofinterest, by the socially connected users, in the particular entity willgenerally increase as additional similarities between the sociallyconnected entities and/or the directly interested entities areidentified.

For example, assume that 75% of the socially connected entities expressan interest in a movie in which the star actor is the same star actor inthe prequel from the examples above. In this situation, this similaritybetween the socially connected users will increase the likelihood thatthe socially connected users are interested users for the prequel. Inturn, this increased likelihood can be extended as an indication thatthe likelihood of a particular user being interested in the prequelincreases when the user expresses an interest in movies that star thesame actor. Thus, an interest score for a candidate entity can be based,at least in part, on a portion (e.g., a number or a percentage) of thesocially connected users that have expressed an interest in (e.g., aredirectly interested users for) the candidate entity. For example, theinterest score for a candidate entity can increase with increases to theportion of the socially connected users that have expressed an interestin the candidate entity.

In some implementations, the interest score for a candidate entity canbe based, at least in part, on a portion of the directly interestedusers for the particular entity that have also expressed an interest in(e.g., are directly interested users for) the candidate entity. Forexample, the interest score for the candidate entity can be increasedbased on a portion of the directly interested users for the particularentity that have expressed an interest in the candidate entity.Increasing the interest score for a candidate entity based on thecandidate entity being of interest to the directly interested users forthe particular entity reflects the increased similarity between theinterests of the directly interested users for the particular entity andthe socially connected users that have expressed an interest in thecandidate entity.

As described above with reference to FIG. 2, the interest score for acandidate entity can also be based on a relationship score for matchingrelationships between the particular entity and the candidate entity, adegree of separation in the knowledge graph between the candidate entityand the particular entity, or other factors that indicate users sharedinterest in the particular entity based on their expressed interest inthe candidate entity.

An expansion entity is selected based on the interest scores (312). Insome implementations, a candidate entity having a highest interest scoreis selected as the expansion entity for the particular entity. In someimplementations, each candidate entity having an interest score thatmeets an interest score threshold is selected as an expansion entity forthe particular entity. The interest score threshold can be expressed,for example, as an absolute interest score (e.g., an interest score of0.5 on a scale from 0.0-1.0). The interest score threshold can also, oralternatively, be expressed relative to other interest scores for theother candidate entities. For example, the interest score threshold canindicate that the candidate entities having interest scores that are ina highest 10% of all interest scores for the candidate entities can beselected as expansion entities for the particular entity.

Additional audience members are selected based on the expansion entity(314). In some implementations, the additional audience members areselected to include users that have expressed an interest in one or moreof the expansion entities. The additional audience members need not be,but can be, socially connected to the directly interested users. Forexample, a user that has expressed an interest in the expansion entity,but not expressed an interest in the particular entity can be selectedas an additional audience member irrespective of whether the user issocially connected to a directly interested user for the particularentity. In some implementations, the set of additional audience membersthat are selected is a disjoint set of users relative to the set ofdirectly interested users for the particular entity.

The number of additional audience members that are selected can be basedon the desired reach that was specified in the request to distributecontent and/or a number of directly interested users for the particularentity. For example, assume that the desired reach specified in therequest to distribute content is 1,000,000 users. Further assume that800,000 directly interested users were identified for the particularentity. In this example, 200,000 additional audience members can beselected to meet the desired reach for the content.

In some implementations, users are selected to be additional audiencemembers based, in part, on the interest scores of the expansion entitiesin which the users have expressed an interest. For example, additionalaudience members can first be selected based on an expansion entityhaving a highest interest score relative to the particular entity. Ifthe desired reach is not met after selecting additional audience membersusing the highest scoring expansion entity, additional audience memberscan be selected using a next highest scoring expansion entity (e.g., theexpansion entity having the second highest interest score).

To illustrate and continuing with the example above in which 200,000additional audience members are needed to meet the desired reach of1,000,000 users, assume that two expansion entities have been identifiedfor the particular entity. Also assume that 150,000 users expressed aninterest in a first expansion entity having an interest score of 0.7 andthat 100,000 users expressed an interest in a second expansion entityhaving an interest score of 0.6. In this example, the 150,000 users thatexpressed an interest in the first candidate entity may be selected asadditional audience members, while 50,000 users that expressed aninterest in the second expansion entity can be selected as additionalaudience members.

Other additional audience member selection techniques can be used. Forexample, for each of the expansion entities, the portion of the usersthat are selected as additional audience members can proportionate to(or a function of) the relative values of the interest scores for theexpansion entities. Continuing with the example above, if the portion ofadditional audience members selected from each of the expansion entitiesis proportional to the interest score, 107,642 additional audiencemembers can be users that expressed an interest in the first expansionentity (e.g., 200,000*0.7/(0.7+0.6)), and 92,308 additional audiencemembers can be users that expressed an interest in the second expansionentity (e.g., 200,000*0.6/(0.6+0.7)).

The content is provided to the users that expressed an interest in theparticular entity and at least a portion of the additional audiencemembers (316). In some implementations, the content is an onlineadvertisement, a music file, a video file, or another portion ofcontent. The content can be distributed for example, in response to arequest to provide content for presentation to a user that has expressedan interest in the particular entity and/or one of the expansionentities. The request can be received, for example, when a directlyinterested user or a user that has been selected as an additionalaudience member requests a social network page or another web page thatincludes a content item slot.

Performance measures for the expansion entities are received (318). Insome implementations, the performance measures for each expansion entityare based, at least in part, on a click-through-rate and/or conversionrate for content provided to additional audience members that wereselected based on their interest in the expansion entity, which arereferred to as additional audience members for the expansion entity. Theclick-through-rate for an expansion entity can be, for example,expressed as a ratio of a number of clicks on content provided to theadditional audience members for the expansion entity relative to anumber of the additional audience members for the expansion entity thatreceived the content. Similarly, a conversion rate for an expansionentity can be, for example, expressed as a ratio of a number ofconversions performed by the additional audience members for theexpansion entity relative to a number of the additional audience membersfor the expansion entity that received the content.

The interest scores for the expansion entities are updated based on theperformance measures (320). In some implementations, the interest scorefor each expansion entity can be scaled based on the performancemeasures for that expansion entity. To illustrate and continuing withthe example above in which the first expansion entity has an interestscore of 0.7 and the second expansion entity has an interest score of0.6, assume that the first expansion entity has a click-through-rate of0.2 and that the second expansion entity has an interest score of 0.3.In this example, the interest score for the second expansion entity canbe adjusted (e.g., increased) relative to the interest score for thefirst expansion entity to reflect the higher click-through-rate for thesecond expansion entity.

The adjustment can be performed, for example, by computing a product ofthe interest score and the performance measures or selecting a scalingfactor based on the performance scores. The scaling factor can be avalue between 0.8 and 1.2 that is selected based on the level ofperformance for an expansion entity relative to the average performanceof all expansion entities that were selected for a particular entity.For example, a scaling factor of 1.0 can be selected for expansionentities having a performance measure that is within a specified amountof the mean performance of the other expansion entities (or allexpansion entities). Similarly, a scaling factor greater than 1.0 can beselected for expansion entities having a performance measure that isabove the specified amount of the mean performance, and a scaling factorless than 1.0 can be selected for expansion entities having performancemeasures below the specified amount of the mean performance. In turn,the interest score for each expansion entity can be scaled, for example,by computing a product (or another function) of the interest score forthe expansion entity and the scaling factor for the expansion entity.

Expansion entity selection can again be performed (312). In someimplementations, the expansion entity selection can be performed usingthe updated interest scores for the expansion entities. The expansionentity selection can be performed in a manner similar to that describedabove.

FIG. 4 is a block diagram of another example process 400 for selectingaudience members for a content item. The process 400 can be performed byone or more data processing apparatus, such as the audience selectionapparatus 120 and/or the content distribution system 110 of FIG. 1.Operations of the process 400 can by execution of instructionsnon-transitory computer readable medium that cause one or more dataprocessing apparatus to perform operations of the process 400.

A set of candidate entities is identified (402). The set of candidateentities can be identified, for example, in a manner similar to thatdescribed above with reference to FIG. 3. For example, either inresponse to receiving a request to distribute content to an audience ofusers that have been deemed interested in a particular entity (or priorto such a request), users that are directly interested users for theparticular entity are identified. Using the social graph, users that aresocially connected to the directly interested users are identified, andusing the knowledge grapy, entities in which the socially connectedusers have expressed an interest can be identified as candidateentities.

Matching relationships between the set of candidate entities and theparticular entity are identified based on the knowledge graph (404). Insome implementations, a matching relationship between a candidate entityand the particular entity is determined to exist when each of thecandidate entity and the particular entity share a same or similarrelationship with a third entity.

For example, with reference to FIG. 2 and according to the knowledgegraph 214, the Prequel 224, MovieA 230, MovieB 232, and MovieC 234 areeach connected to DirectorA 228 by a relationship path labeled “DirectedBy”, and therefore share the relationship “Directed By” with DirectorA228. Thus, in this example, the relationship “Directed By DirectorA” isa matching relationship between the Prequel 222 (the particular entity)and each of the candidate entities MovieA 230, MovieB 232, and MovieC234.

Relationship scores are determined for the matching relationships (406).As noted above, a relationship score is a measure of interest, e.g., bythe set of connected users, in candidate entities having a particularrelationship that is shared by the particular entity and the candidateentities. In some implementations, the relationship score for aparticular relationship is determined based, at least in part, on aportion of the socially connected users that have expressed an interestin an entity having the matching relationship and/or a number of thecandidate entities that have the matching relationship.

For example, with reference to FIG. 2, the determination of therelationship score for “directed by DirectorA” can include identifying aportion of the socially connected users that are directly interestedusers for an entity having the relationship “directed by” with DirectorA228. In this example, each of UserC 218, UserD 208, and UserE 216 haveexpressed an interest in an entity having the relationship “directed by”with DirectorA 228, such that all of the socially connected users forUserA 206 and UserB 210 are directly interested users for an entityhaving the relationship “directed by” with DirectorA 228. In someimplementations, the relationship score could be set to 1.0 indicatingthat 100% of the socially connected users have expressed an interest inan entity having the matching relationship. In some implementations, therelationship score can be increase by a set amount (e.g., 0.1) for eachsocially connected user that has expressed an interest in an entityhaving the matching relationship. Thus, in some implementations, therelationship score for a particular matching relationship will increasewith increases to the portion of the socially connected users that haveexpressed an interest in an entity having the matching relationship.

The determination of the relationship score can also (or alternatively)be based on the portion (e.g., a number of or percentage) of thematching relationship. For example, the determination of therelationship score for “directed by DirectorA” can include identifying aportion of the candidate entities that have the matching relationship.As noted above, MovieA 230, MovieB 232, and Movie C 234 each have thematching relationship “Directed By DirectorA,” but ActressA 236 does nothave the matching relationship. Thus, in this example, 3 of the 4candidate entities (excluding candidate entity DirectorA 228, which isthe subject of the matching relationship) have the matchingrelationship. In some implementations, the relationship score can be setto (or adjusted using) a value, such as 0.75, indicating that 3 out ofthe 4 candidate entities have the matching relationship. In someimplementations, the relationship score can be increased a set amountbased on each of the candidate entities have the matching relationship.Thus, in some implementations, the relationship score for a matchingrelationship will increase with increases in the portion of candidateentities having the matching relationship.

An expansion relationship is selected based on the relationship scores(408). In some implementations, the matching relationship having ahighest relationship score is selected as the expansion relationship forthe particular entity. In some implementations, each matchingrelationship having a relationship score that meets a relationship scorethreshold is selected as an expansion relationship for the particularentity. The relationship score threshold can be expressed, for example,as an absolute relationship score threshold (e.g., a relationship scoreof 0.5 on a scale from 0.0-1.0). The relationship score threshold canalso, or alternatively, be expressed relative to other relationshipscores for the other matching relationships. For example, therelationship score threshold can indicate that the matchingrelationships having relationship scores that are in a highest 10% ofall relationship scores for the matching relationships can be selectedas expansion relationships for the particular entity.

Additional audience members are selected based on the expansionrelationship (410). In some implementations, users are selected to beadditional audience members based, in part, on the relationship scoresof the matching relationships. For example, those users that haveexpressed an interest in entities having a matching relationship with ahighest relationship score can be selected as additional audiencemembers. If the desired reach is not met after selecting additionalaudience members based on the highest relationship score, users thathave expressed an interest in entities having a next highestrelationship score can be selected as additional audience members. Thisselection process can iteratively repeat until the desired reach is met.

Other audience member selection techniques can be used. For example, foreach matching relationship, the portion of the users that are selectedas additional audience members can proportionate to (or a function of)the relative values of the relationship scores. For example, assume that200,000 additional audience members are needed to reach the desiredreach and that a first matching relationship has a relationship score of0.6, while a second matching relationship has a relationship score of0.4. In this example, if the portion of additional audience membersselected using each matching relationship is proportional to therelationship scores, 120,000 (e.g., 200,000*0.6/(0.6+0.4)) users thatexpressed an interest in entities having the first matching relationshipwill be selected as additional audience members, while 80,000 (e.g.,200,000*0.4/(0.4+0.6)) users that expressed an interest in entitieshaving the second matching relationship will be selected as additionalaudience members.

A content item is provided to at least a portion of the additionalaudience members (412). Content items can be provided to the additionalaudience members in a manner similar to that described above withreference to FIG. 3.

Performance measures for the expansion relationships are received (414).In some implementations, the performance measures for each expansionrelationship are based, at least in part, on a click-through-rate and/orconversion rate for content provided to additional audience membersadditional audience members for the expansion relationship (e.g., theadditional audience members that were selected based on the expansionrelationship). The click-through-rate for an expansion relationship canbe, for example, expressed as a ratio of a number of clicks on contentprovided to the additional audience members for the expansionrelationship relative to a number of the additional audience members forthe expansion relationship that received the content. Similarly, aconversion rate for an expansion relationship can be, for example,expressed as a ratio of a number of conversions performed by theadditional audience members for the expansion relationship relative to anumber of the additional audience members for the expansion relationshipthat received the content.

Relationship scores for the expansion relationships are updated based onthe performance measures (416). In some implementations, therelationship score for each expansion relationship can be scaled basedon the performance measures for that expansion relationship. Toillustrate and continuing with the example above in which the firstexpansion relationship has a relationship score of 0.6 and the secondexpansion relationship has a relationship score of 0.4, assume that thefirst expansion relationship has a click-through-rate of 0.2 and thatthe second expansion relationship has an interest score of 0.3. In thisexample, the relationship score for the second expansion relationshipcan be adjusted (e.g., increased) relative to the relationship score forthe first expansion relationship to reflect the higherclick-through-rate for the second expansion relationship.

The adjustment can be performed, for example, by computing a product ofthe relationship score and the performance measures or selecting ascaling factor based on the performance scores. The scaling factor canbe a value (e.g., between 0.8 and 1.2 or a value from another scale)that is selected based on the level of performance for an expansionrelationship relative to the average performance of all expansionrelationships that were selected for a particular entity. For example, ascaling factor of 1.0 can be selected for expansion relationships havinga performance measure that is within a specified amount of the meanperformance of the other expansion relationships (or all expansionrelationships). Similarly, a scaling factor greater than 1.0 can beselected for expansion relationships having a performance measure thatis above the specified amount of the mean performance, and a scalingfactor less than 1.0 can be selected for expansion relationships havingperformance measures below the specified amount of the mean performance.In turn, the interest score for each expansion relationship can bescaled, for example, by computing a product (or another function) of therelationship score for the expansion relationship and the scaling factorfor the expansion relationship.

An expansion relationship can again be performed (408). In someimplementations, the expansion relationship selection can be performedusing the updated relationship scores for the expansion relationships.The expansion relationship selection can be performed in a mannersimilar to that described above.

FIG. 5 is a flow chart of an example process 500 for distributingcontent based on an entity bid. The process 500 can be performed by oneor more data processing apparatus, such as the audience selectionapparatus 120 and/or the content distribution system 110 of FIG. 1.Operations of the process 500 can be implemented by execution ofinstructions stored on a non-transitory computer readable medium thatcause one or more data processing apparatus to perform operations of theprocess 500.

Bid data are received for a content item (502). In some implementations,the bid data are received from a data store storing the bid data, forexample, with a reference to the content item. The bid data can bereceived, for example, from a content item provider that has requesteddistribution of the content item.

The bid data can include, for example, a bid and a bid entity. The bidis a specified value that the content item provider will pay fordistribution of the content item. For example, the bid can be a maximumamount that the content item provider will pay for each presentation ofand/or interaction with the content item.

The bid entity is data specifying one or more entities with whichvalidity of the bid is conditioned. In some implementations, the bidentity specifies that the bid is only valid for presentations of thecontent item to users that have been deemed interested in at least oneof the specified one or more entities. For example, assume that the biddata includes a bid value of $1.00 cost-per-click, and that the bidvalue is conditioned on the content item is presented to users that havebeen deemed interested in the prequel from the examples above. In thisexample, the content item provider will pay up to $1.00 for each clickof the content item by users that have been deemed interested in theprequel, but in this example, has not agreed to pay for presentations ofor clicks on the content item by users that have not been deemedinterested in the prequel.

The bid data can also include, for example, a desired reach specifying anumber of users to whom the content item is to be presented. Forexample, the desired reach can indicate that the content item is to bepresented to 1,000,000 users that have been deemed interested in theprequel from the example above (or another particular entity).

In some implementations, the bid data can specify a different bid foreach of multiple different entities, such that a content item providercan control the amount spent for distribution of the content item basedon the entities in which the users are interested. For example, anadvertiser can specify a bid of $100 per thousand impressions (e.g.,$100 CPM) for presentations of an advertisement to users that have beendeemed interested (e.g., directly interested) in the prequel, and canspecify a bid of $10 CPM for presentations of the advertisement to usersthat have been deemed interested in a spoof of the prequel. Thus, theadvertiser can specify different amounts that they are willing to paybased on the interests of the users.

A determination is made whether the desired reach will be met by usersthat are directly interested users for the bid entity (504). In someimplementations, the determination can be made by identifying a numberof users that have affirmatively expressed an interest in the bid entityand determining whether the identified number of users meets (e.g.,equals or exceeds) the desired reach. As noted above, more than one bidentity can be specified, such that the determination can be made basedon a total number of users that have expressed an interest in any of thebid entities.

If the desired reach is met, the content item can be distributed basedon the bid value that was specified for (e.g., associated with) the bidentity (506). Continuing with the example above, the advertisement canbe distributed to directly interested users for the prequel at a cost of(or based on a cost of) $100 CPM, while the advertisement can bedistributed to directly interested users for the spoof of the prequelfor $10 CPM.

If the desired reach is not met, an expansion entity can be identifiedfor the bid entity (508). The expansion entity for the bid entity can beidentified, for example, in a manner similar to that described abovewith reference to FIGS. 1-4.

An expansion bid value is determined based on the bid data and theexpansion entity (510). The expansion bid value is a value that acontent item provider will pay for distribution of a content item toadditional audience members that were identified based on an expansionentity and/or an expansion relationship. The expansion bid value can bedetermined based on a function of the bid value, the interest score foran expansion entity, the relationship score for an expansionrelationship, a degree of separation between the bid entity and theexpansion entity, or other information indicative of a level of interestof the additional audience members in the bid entity.

For example, assume that a particular expansion entity has an interestscore (relative to the bid entity) of 0.7, indicating that there is a70% likelihood that the additional audience members selected using theparticular expansion entity are interested in the bid entity. In thisexample, the expansion bid value can be determined to be the product (oranother function) of the bid value and the interest score for theparticular expansion entity. Thus, in this example, the expansion bidvalue will be 70% of the bid value for the bid entity.

When an expansion relationship is used to identify additional audiencemembers, the expansion bid value can be determined in a similar manner.For example, the bid value for the bid entity can be adjusted using therelationship score for the expansion relationship that was used toidentify the additional audience members. To illustrate, assume that therelationship score for a particular expansion relationship is 0.5, andthat additional audience members are selected based on this expansionrelationship. In this example, the expansion bid value can be 50% of thebid value for the bid entity. Other bid scaling techniques can also beused, for example, based on specified relationships between relationshipscores (or interest scores) and bid adjustment factors. Thus, theexpansion bid value need not be directly proportional to either theinterest scores or relationship scores.

In some implementations, the expansion bid value can be based, at leastin part, on a degree of separation between the bid entity and anexpansion entity for the bid entity. For example, the expansion bidvalue can be decreased by a set amount for each degree of separation(e.g., for each relationship path) between the bid entity and the entityin which the additional audience members expressed an interest. Toillustrate and with reference to FIG. 2, assume that the bid entity isthe prequel 224 and that the additional audience members were identifiedusers that are directly interested users for MovieC 234. In thisexample, there are two degrees of separation between the prequel 224 andMovieC 234, such that the expansion bid value can be decreased by twotimes the amount specified for a degree of separation. In someimplementations, degree of separation adjustments can be combined withother techniques for determining the expansion bid value.

The content item is distributed based on the expansion bid value (512).In some implementations, the content item is distributed to theadditional audience members that were selected for the content item, andthe price charged to the content item provider will be the expansion bidvalue (or based on the expansion bid value) corresponding to theexpansion entity or expansion relationship that was used to select theadditional audience members.

FIG. 6 is block diagram of an example computer system 600 that can beused to perform operations described above. The system 600 includes aprocessor 610, a memory 620, a storage device 630, and an input/outputdevice 640. Each of the components 610, 620, 630, and 640 can beinterconnected, for example, using a system bus 650. The processor 610is capable of processing instructions for execution within the system600. In one implementation, the processor 610 is a single-threadedprocessor. In another implementation, the processor 610 is amulti-threaded processor. The processor 610 is capable of processinginstructions stored in the memory 620 or on the storage device 630.

The memory 620 stores information within the system 600. In oneimplementation, the memory 620 is a computer-readable medium. In oneimplementation, the memory 620 is a volatile memory unit. In anotherimplementation, the memory 620 is a non-volatile memory unit.

The storage device 630 is capable of providing mass storage for thesystem 600. In one implementation, the storage device 630 is acomputer-readable medium. In various different implementations, thestorage device 630 can include, for example, a hard disk device, anoptical disk device, a storage device that is shared over a network bymultiple computing devices (e.g., a cloud storage device), or some otherlarge capacity storage device.

The input/output device 640 provides input/output operations for thesystem 600. In one implementation, the input/output device 640 caninclude one or more of a network interface devices, e.g., an Ethernetcard, a serial communication device, e.g., and RS-232 port, and/or awireless interface device, e.g., and 802.11 card. In anotherimplementation, the input/output device can include driver devicesconfigured to receive input data and send output data to otherinput/output devices, e.g., keyboard, printer and display devices 660.Other implementations, however, can also be used, such as mobilecomputing devices, mobile communication devices, set-top box televisionclient devices, etc.

Although an example processing system has been described in FIG. 6,implementations of the subject matter and the functional operationsdescribed in this specification can be implemented in other types ofdigital electronic circuitry, or in computer software, firmware, orhardware, including the structures disclosed in this specification andtheir structural equivalents, or in combinations of one or more of them.

Embodiments of the subject matter and the operations described in thisspecification can be implemented in digital electronic circuitry, or incomputer software, firmware, or hardware, including the structuresdisclosed in this specification and their structural equivalents, or incombinations of one or more of them. Embodiments of the subject matterdescribed in this specification can be implemented as one or morecomputer programs, i.e., one or more modules of computer programinstructions, encoded on computer storage medium for execution by, or tocontrol the operation of, data processing apparatus. Alternatively or inaddition, the program instructions can be encoded on anartificially-generated propagated signal, e.g., a machine-generatedelectrical, optical, or electromagnetic signal, that is generated toencode information for transmission to suitable receiver apparatus forexecution by a data processing apparatus. A computer storage medium canbe, or be included in, a computer-readable storage device, acomputer-readable storage substrate, a random or serial access memoryarray or device, or a combination of one or more of them. Moreover,while a computer storage medium is not a propagated signal, a computerstorage medium can be a source or destination of computer programinstructions encoded in an artificially-generated propagated signal. Thecomputer storage medium can also be, or be included in, one or moreseparate physical components or media (e.g., multiple CDs, disks, orother storage devices).

The operations described in this specification can be implemented asoperations performed by a data processing apparatus on data stored onone or more computer-readable storage devices or received from othersources.

The term “data processing apparatus” encompasses all kinds of apparatus,devices, and machines for processing data, including by way of example aprogrammable processor, a computer, a system on a chip, or multipleones, or combinations, of the foregoing The apparatus can includespecial purpose logic circuitry, e.g., an FPGA (field programmable gatearray) or an ASIC (application-specific integrated circuit). Theapparatus can also include, in addition to hardware, code that createsan execution environment for the computer program in question, e.g.,code that constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, a cross-platform runtimeenvironment, a virtual machine, or a combination of one or more of them.The apparatus and execution environment can realize various differentcomputing model infrastructures, such as web services, distributedcomputing and grid computing infrastructures.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, object, orother unit suitable for use in a computing environment. A computerprogram may, but need not, correspond to a file in a file system. Aprogram can be stored in a portion of a file that holds other programsor data (e.g., one or more scripts stored in a markup languagedocument), in a single file dedicated to the program in question, or inmultiple coordinated files (e.g., files that store one or more modules,sub-programs, or portions of code). A computer program can be deployedto be executed on one computer or on multiple computers that are locatedat one site or distributed across multiple sites and interconnected by acommunication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform actions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. The essential elements of a computer area processor for performing actions in accordance with instructions andone or more memory devices for storing instructions and data. Generally,a computer will also include, or be operatively coupled to receive datafrom or transfer data to, or both, one or more mass storage devices forstoring data, e.g., magnetic, magneto-optical disks, or optical disks.However, a computer need not have such devices. Moreover, a computer canbe embedded in another device, e.g., a mobile telephone, a personaldigital assistant (PDA), a mobile audio or video player, a game console,a Global Positioning System (GPS) receiver, or a portable storage device(e.g., a universal serial bus (USB) flash drive), to name just a few.Devices suitable for storing computer program instructions and datainclude all forms of non-volatile memory, media and memory devices,including by way of example semiconductor memory devices, e.g., EPROM,EEPROM, and flash memory devices; magnetic disks, e.g., internal harddisks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROMdisks. The processor and the memory can be supplemented by, orincorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's client device in response to requests received from the webbrowser.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back-end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front-end component, e.g., aclient computer having a graphical user interface or a Web browserthrough which a user can interact with an implementation of the subjectmatter described in this specification, or any combination of one ormore such back-end, middleware, or front-end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a local area network (“LAN”) and a widearea network (“WAN”), an inter-network (e.g., the Internet), andpeer-to-peer networks (e.g., ad hoc peer-to-peer networks).

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someembodiments, a server transmits data (e.g., an HTML page) to a clientdevice (e.g., for purposes of displaying data to and receiving userinput from a user interacting with the client device). Data generated atthe client device (e.g., a result of the user interaction) can bereceived from the client device at the server.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinventions or of what may be claimed, but rather as descriptions offeatures specific to particular embodiments of particular inventions.Certain features that are described in this specification in the contextof separate embodiments can also be implemented in combination in asingle embodiment. Conversely, various features that are described inthe context of a single embodiment can also be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

Thus, particular embodiments of the subject matter have been described.Other embodiments are within the scope of the following claims. In somecases, the actions recited in the claims can be performed in a differentorder and still achieve desirable results. In addition, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In certain implementations, multitasking and parallelprocessing may be advantageous.

What is claimed is:
 1. A method performed by one or more data processingapparatus, wherein the one or more data processing apparatus compriseone or more processors configured to perform operations comprising:receiving a request to distribute content to a specified number of usersof a particular social network that are interested in a first entity;determining that distribution of the content to users of the particularsocial network that have expressed an interest in the first entity willreach fewer than the specified number of users; and in response todetermining that the distribution will reach fewer than the specifiednumber of users, using links in both of a social graph and a knowledgegraph to expand an audience for the content beyond the users that haveexpressed an interest in the first entity, including: identifying,within a social graph, links between nodes that identify a set of usersthat are socially connected, in the particular social network, to aparticular user who has expressed an interest in the first entity; afteridentifying the set of users, identifying a set of entities beyond thefirst entity in which the set of users have expressed an interest;accessing the knowledge graph to identify one or more relationships thatconnect the set of entities to the first entity in the knowledge graph;determining, based on the identified relationships, a largest number ofentities from the set of entities that share a same relationship withthe first entity; classifying the same relationship to the first entityas an expansion relationship based on the largest number of entitiessharing the same relationship with the first entity; selecting the usersin the particular social network that have expressed an interest inother entities that share the same relationship with the first entity,but have not expressed an interest in the first entity as additionalaudience members that will receive the content based on the samerelationship being classified as an expansion relationship for the firstentity; and distributing the content to the users of the particularsocial network that have expressed an interest in the first entity aswell as the additional audience members that have expressed an interestin the other entities.
 2. The method of claim 1, further comprisingdetermining a relationship score for the same relationship based on aportion of the set of users who have expressed an interest in an entitythat shares the same relationship with the first entity, whereinclassifying the same relationship as an expansion relationship isfurther based on the relationship score.
 3. The method of claim 2,wherein determining the relationship score further comprises determiningthe relationship score based on a portion of the set of entities thatshare the same relationship with the first entity.
 4. The method ofclaim 2, further comprising: classifying one or more additionalrelationships between the first entity and entities in the set ofentities as an expansion relationship based on correspondingrelationship scores for the one or more additional relationships meetinga relationship score threshold; and distributing the content to users ofthe particular social network that have expressed an interest in the oneor more additional entities, but have not expressed an interest in thefirst entity.
 5. The method of claim 1, wherein determining thatdistribution of the content to users of the particular social networkthat have expressed an interest in the first entity will reach fewerthan the specified number of users comprises determining that fewer ofthan the specified number of users have interacted with a positivefeedback element that references the first entity.
 6. The method ofclaim 5, wherein determining that fewer of than the specified number ofusers have interacted with a positive feedback element that referencesthe first entity comprises determining that fewer of than the specifiednumber of users have performed none of interacting with a positivefeedback element that references the first entity, visiting a socialnetwork page about the first entity, and identifying the first entity asan interest in the users' social network profiles.
 7. The method ofclaim 1, wherein selecting the users as additional audience memberscomprises selecting, as additional audience members, only those usersthat are socially connected to the particular user in the particularsocial network and that have expressed an interest in at least one ofthe other entities.
 8. A system, comprising: one or more data storesstoring data representing a social graph and data representing aknowledge graph that is different from the social graph; and one or moredata processing apparatus comprising one or more processors, wherein theone or more processors interact with the one or more data stores andexecute instructions that cause the one or more data processingapparatus to perform operations comprising: receiving a request todistribute content to a specified number of users of a particular socialnetwork that are interested in a first entity; determining thatdistribution of the content to users of the particular social networkthat have expressed an interest in the first entity will reach fewerthan the specified number of users; and in response to determining thatthe distribution will reach fewer than the specified number of users,using links in both of a social graph and a knowledge graph to expand anaudience for the content beyond the users that have expressed aninterest in the first entity, including: identifying, within a socialgraph, links between nodes that identify a set of users that aresocially connected, in the particular social network, to a particularuser who has expressed an interest in the first entity; afteridentifying the set of users, identifying a set of entities beyond thefirst entity in which the set of users have expressed an interest;accessing the knowledge graph to identify one or more relationships thatconnect the set of entities to the first entity in the knowledge graph;determining, based on the identified relationships, a largest number ofentities from the set of entities that share a same relationship withthe first entity; classifying the same relationship to the first entityas an expansion relationship based on the largest number of entitiessharing the same relationship with the first entity; selecting the usersin the particular social network that have expressed an interest inother entities that share the same relationship with the first entity,but have not expressed an interest in the first entity as additionalaudience members that will receive the content based on the samerelationship being classified as an expansion relationship for the firstentity; and distributing the content to the users of the particularsocial network that have expressed an interest in the first entity aswell as the additional audience members that have expressed an interestin the other entities.
 9. The system of claim 8, wherein theinstructions cause the one or more data processing apparatus to performoperations comprising determining a relationship score for the samerelationship based on a portion of the set of users who have expressedan interest in an entity that shares the same relationship with thefirst entity, wherein classifying the same relationship as an expansionrelationship is further based on the relationship score.
 10. The systemof claim 9, wherein determining the relationship score further comprisesdetermining the relationship score based on a portion of the set ofentities that share the same relationship with the first entity.
 11. Thesystem of claim 9, wherein the instructions cause the one or more dataprocessing apparatus to perform operations comprising: classifying oneor more additional relationships between the first entity and entitiesin the set of entities as an expansion relationship based oncorresponding relationship scores for the one or more additionalrelationships meeting a relationship score threshold; and distributingthe content to users of the particular social network that haveexpressed an interest in the one or more additional entities, but havenot expressed an interest in the first entity.
 12. The system of claim8, wherein determining that distribution of the content to users of theparticular social network that have expressed an interest in the firstentity will reach fewer than the specified number of users comprisesdetermining that fewer of than the specified number of users haveinteracted with a positive feedback element that references the firstentity.
 13. The system of claim 12, wherein determining that fewer ofthan the specified number of users have interacted with a positivefeedback element that references the first entity comprises determiningthat fewer of than the specified number of users have performed none ofinteracting with a positive feedback element that references the firstentity, visiting a social network page about the first entity, andidentifying the first entity as an interest in the users' social networkprofiles.
 14. The system of claim 8, wherein selecting the users asadditional audience members comprises selecting, as additional audiencemembers, only those users that are socially connected to the particularuser in the particular social network and that have expressed aninterest in at least one of the other entities.
 15. A non-transitorycomputer storage medium encoded with a computer program, the programcomprising instructions that when executed by one or more dataprocessing apparatus cause the one or more data processing apparatus toperform operations comprising: receiving a request to distribute contentto a specified number of users of a particular social network that areinterested in a first entity; determining that distribution of thecontent to users of the particular social network that have expressed aninterest in the first entity will reach fewer than the specified numberof users; and in response to determining that the distribution willreach fewer than the specified number of users, using links in both of asocial graph and a knowledge graph to expand an audience for the contentbeyond the users that have expressed an interest in the first entity,including: identifying, within a social graph, links between nodes thatidentify a set of users that are socially connected, in the particularsocial network, to a particular user who has expressed an interest inthe first entity; after identifying the set of users, identifying a setof entities beyond the first entity in which the set of users haveexpressed an interest; accessing the knowledge graph to identify one ormore relationships that connect the set of entities to the first entityin the knowledge graph; determining, based on the identifiedrelationships, a largest number of entities from the set of entitiesthat share a same relationship with the first entity; classifying thesame relationship to the first entity as an expansion relationship basedon the largest number of entities sharing the same relationship with thefirst entity; selecting the users in the particular social network thathave expressed an interest in other entities that share the samerelationship with the first entity, but have not expressed an interestin the first entity as additional audience members that will receive thecontent based on the same relationship being classified as an expansionrelationship for the first entity; and distributing the content to theusers of the particular social network that have expressed an interestin the first entity as well as the additional audience members that haveexpressed an interest in the other entities.
 16. The non-transitorycomputer storage medium of claim 15, wherein the instructions cause theone or more data processing apparatus to perform operations comprisingdetermining a relationship score for the same relationship based on aportion of the set of users who have expressed an interest in an entitythat shares the same relationship with the first entity, whereinclassifying the same relationship as an expansion relationship isfurther based on the relationship score.
 17. The non-transitory computerstorage medium of claim 16, wherein determining the relationship scorefurther comprises determining the relationship score based on a portionof the set of entities that share the same relationship with the firstentity.
 18. The non-transitory computer storage medium of claim 16,wherein the instructions cause the one or more data processing apparatusto perform operations comprising: classifying one or more additionalrelationships between the first entity and entities in the set ofentities as an expansion relationship based on correspondingrelationship scores for the one or more additional relationships meetinga relationship score threshold; and distributing the content to users ofthe particular social network that have expressed an interest in the oneor more additional entities, but have not expressed an interest in thefirst entity.
 19. The non-transitory computer storage medium of claim15, wherein determining that distribution of the content to users of theparticular social network that have expressed an interest in the firstentity will reach fewer than the specified number of users comprisesdetermining that fewer of than the specified number of users haveinteracted with a positive feedback element that references the firstentity.
 20. The non-transitory computer storage medium of claim 19,wherein determining that fewer of than the specified number of usershave interacted with a positive feedback element that references thefirst entity comprises determining that fewer of than the specifiednumber of users have performed none of interacting with a positivefeedback element that references the first entity, visiting a socialnetwork page about the first entity, and identifying the first entity asan interest in the users' social network profiles.